<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
<meta name="generator" content="Doxygen 1.12.0"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>NeuZephyr: D:/Users/Mgepahmge/Documents/C Program/NeuZephyr/include/NeuZephyr/Optimizer.cuh File Reference</title>
<link rel="icon" href="NZ_logo2.png" type="image/x-icon" />
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr id="projectrow">
  <td id="projectlogo"><img alt="Logo" src="NZ_logo2.png"/></td>
  <td id="projectalign">
   <div id="projectname">NeuZephyr
   </div>
   <div id="projectbrief">Simple DL Framework</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.12.0 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
$(function() { codefold.init(0); });
/* @license-end */
</script>
  <div id="navrow1" class="tabs">
    <ul class="tablist">
      <li><a href="index.html"><span>Main&#160;Page</span></a></li>
      <li><a href="pages.html"><span>Related&#160;Pages</span></a></li>
      <li><a href="namespaces.html"><span>Namespaces</span></a></li>
      <li><a href="annotated.html"><span>Classes</span></a></li>
      <li class="current"><a href="files.html"><span>Files</span></a></li>
    </ul>
  </div>
  <div id="navrow2" class="tabs2">
    <ul class="tablist">
      <li><a href="files.html"><span>File&#160;List</span></a></li>
    </ul>
  </div>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
$(function(){ initResizable(false); });
/* @license-end */
</script>
<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="dir_d522931ffa1371640980b621734a4381.html">Users</a></li><li class="navelem"><a class="el" href="dir_a7e6ee1ae3f772c9504a0b543f2027e2.html">Mgepahmge</a></li><li class="navelem"><a class="el" href="dir_e03f57e346cc4845a4c354a35630b169.html">Documents</a></li><li class="navelem"><a class="el" href="dir_231a0482af2b83c895f27ba7fe745141.html">C Program</a></li><li class="navelem"><a class="el" href="dir_0fa7fc3a0dfd304dbfc9dce9f6facfa2.html">NeuZephyr</a></li><li class="navelem"><a class="el" href="dir_e7295b03dab2e9cdf32139bd8ec2e607.html">include</a></li><li class="navelem"><a class="el" href="dir_657344ecc65cfc28732701509f8d8421.html">NeuZephyr</a></li>  </ul>
</div>
</div><!-- top -->
<div id="doc-content">
<div class="header">
  <div class="summary">
<a href="#nested-classes">Classes</a> &#124;
<a href="#namespaces">Namespaces</a>  </div>
  <div class="headertitle"><div class="title">Optimizer.cuh File Reference</div></div>
</div><!--header-->
<div class="contents">

<p>Definition of optimization algorithms for training deep learning models.  
<a href="#details">More...</a></p>
<div class="textblock"><code>#include &lt;unordered_map&gt;</code><br />
<code>#include &quot;<a class="el" href="_nodes_8cuh_source.html">Nodes.cuh</a>&quot;</code><br />
</div><div class="textblock"><div class="dynheader">
Include dependency graph for Optimizer.cuh:</div>
<div class="dyncontent">
<div class="center"><img src="_optimizer_8cuh__incl.png" border="0" usemap="#a_d_1_2_users_2_mgepahmge_2_documents_2_c_01_program_2_neu_zephyr_2include_2_neu_zephyr_2_optimizer_8cuh" alt=""/></div>
<map name="a_d_1_2_users_2_mgepahmge_2_documents_2_c_01_program_2_neu_zephyr_2include_2_neu_zephyr_2_optimizer_8cuh" id="a_d_1_2_users_2_mgepahmge_2_documents_2_c_01_program_2_neu_zephyr_2include_2_neu_zephyr_2_optimizer_8cuh">
<area shape="rect" title="Definition of optimization algorithms for training deep learning models." alt="" coords="19,5,222,80"/>
<area shape="rect" title=" " alt="" coords="5,128,116,155"/>
<area shape="poly" title=" " alt="" coords="100,82,79,116,74,113,96,79"/>
<area shape="rect" href="_nodes_8cuh.html" title="Declaration of the Node class and various derived node classes for neural network operations." alt="" coords="140,128,224,155"/>
<area shape="poly" title=" " alt="" coords="146,79,168,113,163,116,141,82"/>
<area shape="rect" title=" " alt="" coords="97,203,165,229"/>
<area shape="poly" title=" " alt="" coords="175,157,151,192,146,189,171,154"/>
<area shape="rect" href="_tensor_8cuh.html" title="Definition of the Tensor class for GPU&#45;based tensor operations." alt="" coords="190,203,275,229"/>
<area shape="poly" title=" " alt="" coords="193,154,217,189,213,192,188,157"/>
<area shape="rect" title=" " alt="" coords="83,277,144,304"/>
<area shape="poly" title=" " alt="" coords="213,232,149,271,146,267,210,228"/>
<area shape="rect" title=" " alt="" coords="105,352,184,379"/>
<area shape="poly" title=" " alt="" coords="227,231,162,340,157,337,223,228"/>
<area shape="rect" title=" " alt="" coords="208,352,265,379"/>
<area shape="poly" title=" " alt="" coords="236,230,238,336,233,337,230,230"/>
<area shape="rect" href="dl__export_8cuh_source.html" title=" " alt="" coords="384,352,484,379"/>
<area shape="poly" title=" " alt="" coords="277,222,335,240,366,256,393,275,415,305,429,336,424,338,411,308,389,279,363,260,333,245,275,227"/>
<area shape="rect" href="_dimension_8cuh_source.html" title=" " alt="" coords="270,277,379,304"/>
<area shape="poly" title=" " alt="" coords="250,228,299,266,295,270,247,232"/>
<area shape="poly" title=" " alt="" coords="294,307,191,348,189,343,292,302"/>
<area shape="poly" title=" " alt="" coords="311,307,265,344,262,340,307,302"/>
<area shape="poly" title=" " alt="" coords="345,302,404,341,401,346,342,307"/>
<area shape="rect" title=" " alt="" coords="289,352,360,379"/>
<area shape="poly" title=" " alt="" coords="327,305,327,336,322,336,322,305"/>
</map>
</div>
</div><div class="textblock"><div class="dynheader">
This graph shows which files directly or indirectly include this file:</div>
<div class="dyncontent">
<div class="center"><img src="_optimizer_8cuh__dep__incl.png" border="0" usemap="#a_d_1_2_users_2_mgepahmge_2_documents_2_c_01_program_2_neu_zephyr_2include_2_neu_zephyr_2_optimizer_8cuhdep" alt=""/></div>
<map name="a_d_1_2_users_2_mgepahmge_2_documents_2_c_01_program_2_neu_zephyr_2include_2_neu_zephyr_2_optimizer_8cuhdep" id="a_d_1_2_users_2_mgepahmge_2_documents_2_c_01_program_2_neu_zephyr_2include_2_neu_zephyr_2_optimizer_8cuhdep">
<area shape="rect" title="Definition of optimization algorithms for training deep learning models." alt="" coords="232,5,434,80"/>
<area shape="rect" href="_compute_graph_8cuh.html" title="Defines the ComputeGraph class for constructing and managing computational graphs in neural network m..." alt="" coords="123,128,325,203"/>
<area shape="poly" title=" " alt="" coords="292,94,259,129,255,126,288,90"/>
<area shape="rect" href="_optimizer_8cu_source.html" title=" " alt="" coords="349,136,538,195"/>
<area shape="poly" title=" " alt="" coords="379,90,420,134,416,137,375,94"/>
<area shape="rect" href="_model_8cuh.html" title="Core class for computational graph construction and neural network modeling." alt="" coords="5,251,208,325"/>
<area shape="poly" title=" " alt="" coords="179,216,144,252,140,248,176,212"/>
<area shape="rect" href="_compute_graph_8cu_source.html" title=" " alt="" coords="232,259,453,317"/>
<area shape="poly" title=" " alt="" coords="273,212,316,256,312,260,269,216"/>
<area shape="rect" href="_model_8cu_source.html" title=" " alt="" coords="22,373,191,432"/>
<area shape="poly" title=" " alt="" coords="109,341,109,373,104,373,104,341"/>
</map>
</div>
</div>
<p><a href="_optimizer_8cuh_source.html">Go to the source code of this file.</a></p>
<table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a id="nested-classes" name="nested-classes"></a>
Classes</h2></td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_optimizer.html">nz::opt::Optimizer</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight">Base class for optimization algorithms in deep learning.  <a href="classnz_1_1opt_1_1_optimizer.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_s_g_d.html">nz::opt::SGD</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight">Stochastic Gradient Descent (<a class="el" href="classnz_1_1opt_1_1_s_g_d.html" title="Stochastic Gradient Descent (SGD) optimizer for deep learning models.">SGD</a>) optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_s_g_d.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_momentum.html">nz::opt::Momentum</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight"><a class="el" href="classnz_1_1opt_1_1_momentum.html" title="Momentum optimizer for deep learning models.">Momentum</a> optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_momentum.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_ada_grad.html">nz::opt::AdaGrad</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight"><a class="el" href="classnz_1_1opt_1_1_ada_grad.html" title="AdaGrad optimizer for deep learning models.">AdaGrad</a> optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_ada_grad.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_r_m_sprop.html">nz::opt::RMSprop</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight"><a class="el" href="classnz_1_1opt_1_1_r_m_sprop.html" title="RMSprop optimizer for deep learning models.">RMSprop</a> optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_r_m_sprop.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_adam.html">nz::opt::Adam</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight"><a class="el" href="classnz_1_1opt_1_1_adam.html" title="Adam optimizer for deep learning models.">Adam</a> optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_adam.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_n_adam.html">nz::opt::NAdam</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight"><a class="el" href="classnz_1_1opt_1_1_n_adam.html" title="NAdam optimizer for deep learning models.">NAdam</a> optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_n_adam.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">class &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classnz_1_1opt_1_1_ada_delta.html">nz::opt::AdaDelta</a></td></tr>
<tr class="memdesc:"><td class="mdescLeft">&#160;</td><td class="mdescRight"><a class="el" href="classnz_1_1opt_1_1_ada_delta.html" title="AdaDelta optimizer for deep learning models.">AdaDelta</a> optimizer for deep learning models.  <a href="classnz_1_1opt_1_1_ada_delta.html#details">More...</a><br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table><table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a id="namespaces" name="namespaces"></a>
Namespaces</h2></td></tr>
<tr class="memitem:"><td class="memItemLeft" align="right" valign="top">namespace &#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="namespacenz_1_1opt.html">nz::opt</a></td></tr>
<tr class="memdesc:namespacenz_1_1opt"><td class="mdescLeft">&#160;</td><td class="mdescRight">Contains optimization algorithms for training deep learning models. <br /></td></tr>
<tr class="separator:"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table>
<a name="details" id="details"></a><h2 class="groupheader">Detailed Description</h2>
<div class="textblock"><p>Definition of optimization algorithms for training deep learning models. </p>
<p>This file declares a set of optimization algorithms designed to update the parameters of deep learning models during training. The algorithms aim to minimize the loss function using different strategies, such as adjusting learning rates dynamically or incorporating momentum to accelerate convergence. Each optimizer implements a <code>step</code> function to perform parameter updates, leveraging GPU-based tensor operations for efficiency.</p>
<p>The optimizers included in this file are:</p><ul>
<li><b>SGD (Stochastic Gradient Descent)</b>: Updates parameters using the negative gradient and a fixed learning rate.</li>
<li><b>Momentum</b>: Improves SGD by adding a momentum term to help smooth out updates and accelerate convergence.</li>
<li><b>AdaGrad</b>: Adjusts learning rates based on historical gradient information, adapting to each parameter's behavior.</li>
<li><b>RMSprop</b>: Uses a moving average of squared gradients to stabilize learning rate adjustments.</li>
<li><b>Adam (Adaptive Moment Estimation)</b>: Combines momentum and RMSprop by computing adaptive learning rates using first and second moment estimates.</li>
<li><b>NAdam (Nesterov-accelerated Adam)</b>: Enhances Adam with Nesterov momentum for improved convergence speed.</li>
<li><b>AdaDelta</b>: An extension of AdaGrad that maintains a running average of squared updates for consistent learning rates.</li>
</ul>
<p>These optimizers are part of the <code><a class="el" href="namespacenz_1_1opt.html" title="Contains optimization algorithms for training deep learning models.">nz::opt</a></code> namespace and are designed for extensibility and high performance in deep learning workflows.</p>
<dl class="section note"><dt>Note</dt><dd>The <code>step</code> function in each optimizer updates model parameters, which are represented by objects compatible with the project's computational graph (e.g., nodes in the graph). Ensure proper memory management and error handling when integrating these optimizers into training processes.</dd></dl>
<dl class="section author"><dt>Author</dt><dd>Mgepahmge(<a href="https://github.com/Mgepahmge">https://github.com/Mgepahmge</a>)</dd></dl>
<dl class="section date"><dt>Date</dt><dd>2024/12/07 </dd></dl>

<p class="definition">Definition in file <a class="el" href="_optimizer_8cuh_source.html">Optimizer.cuh</a>.</p>
</div></div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated by&#160;<a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.12.0
</small></address>
</div><!-- doc-content -->
</body>
</html>
